Skip to content

fix(search)!: fan out a qualified edge into one entry per tuple - #801

Open
ddeboer wants to merge 8 commits into
mainfrom
worktree-weld-fanout
Open

fix(search)!: fan out a qualified edge into one entry per tuple#801
ddeboer wants to merge 8 commits into
mainfrom
worktree-weld-fanout

Conversation

@ddeboer

@ddeboer ddeboer commented Sep 2, 2026

Copy link
Copy Markdown
Member

Fix #798.

A welded co-element filter – this agent in this role – hangs Typesense 30.2 indefinitely whenever the nested entry’s leaves hold arrays, which is every document a qualified relation produced in practice. Through the GraphQL surface that reached a consumer as Unexpected error.

What this changes, and why it is not a workaround

The hang is why this got noticed. It is not why the shape changes.

A weld asks whether one entry satisfies every condition. An entry holding { role: ["etser", "etcher"], creator_id: ["p1", "p3"] } stands for four (role, agent) pairs at once, so it answers the weld with none of them – and the weld silently degenerates into the cross-product it exists to exclude. That entry is not a valid input the engine mishandles; it is a shape that never had a meaning, and one ADR 24’s own words – one entry per edge – already exclude.

So: a weldable nested leaf is single-valued, and multiplicity moves to the entry list.

  • searchSchema refuses a nested field declaring both filterable and array. output-only leaves are untouched – nothing welds them, so they may carry a list for display.
  • The projection emits one entry per combination of an edge’s weldable leaves, splitting the framed node before projection so each leaf still passes through transform, folding and the facet companion unchanged.
  • Nesting is decided by where a node is projected, not by its type – a Root Type reached by a local lookup is nested just as much as a Reference Type.
  • A nested identity companion holds one id per entry where fan-out underwrites that (filterable), and is declared under the arity of its path. That companion is the leaf a weld actually names, so it was the load-bearing half: fan-out alone still left creator_id: ["p1"] inside each entry, which is exactly what hangs 30.2.

Full reasoning, measurements and the rejected alternatives are in the new ADR 26.

Measured

Against live containers, 1 000 000 documents, same data and same query:

welded filter, p50 role alone facet
30.2 + fan-out (this PR) 14–19 ms 15.9 ms 30 ms
31.0 + arrays 13–16 ms 15.8 ms 64 ms
30.2 + arrays (before) hangs 15.8 ms 30 ms
31.0 + fan-out 13–16 ms 16.1 ms 28 ms

The upgrade unwinds nothing: the fanned-out shape is no slower on 31.0 than the array shape it replaces, so when 31.0 goes stable this is routine maintenance rather than a migration back. And facets stay on the real fields: the other shape that works on 30.2 is an index-time composite role|agent key, faster at 3 ms but faceting into 2 142 125 buckets in 1.4 s against 15 buckets in 48 ms.

Cost: indexing is ~15 % slower (54 s against 47 s per 1 000 000 documents) for ~50 % more entries. A batch cost, not a request cost.

No bound, on purpose

An earlier revision capped the entries a document stores (maxEntries, default 100). It has been removed, because it guards the wrong end: by the time the projection runs the CONSTRUCT has matched those values, the endpoint has paid for it, they have crossed the wire, and framing has materialised them into one node. Capping the product declines the last and cheapest step while keeping every expensive one, and bounds nothing framing did not already hold. It also did nothing for the linear case that predates fan-out – a wide edge, or a display-only nesting of ten thousand entries.

The bound belongs where the data enters memory. Capping values per leaf there makes the product k^(weldable leaves), and leaf count is a schema constant – a bound in the schema’s own units rather than a number written against the data’s. That touches a seam serving every consumer of framing, so it is tracked separately, with its own evidence and its own record: #826.

Until then the fan-out is unbounded, exactly as the entry list it replaces always was. What changed is that the growth can now be multiplicative rather than linear.

Defects found in review

Three review rounds found seven real defects, six introduced by this change and none visible from a unit test or the diff. Worth naming, since the fixes are most of the commit history:

  • widening a nested leaf emitted no type at all for a locally-nested Root Type’s facet, which the engine refuses outright;
  • the companion’s declared arity disagreed with what the projection wrote, failing the import for every document whose edge is single-valued;
  • the nesting predicate read a locally-nested Root Type as a root, writing a list where the collection declared one value;
  • narrowing applied to companions earned by facetable alone, which fan-out never splits, dropping ids silently;
  • the entry cap truncated references that could not fan out, spent itself on values that project to nothing, and had no validation.

Every one was caught by running the projection or a live container, which is why welded-filter.integration.test.ts asserts against a real engine rather than a compiled filter string.

Notes for review

  • Breaking: a nested field may no longer declare both filterable and array. Three search-pipeline fixtures carried exactly that shape and are updated here – the same edit a deployment makes.
  • Coverage thresholds move: up in search-typesense (95.69 → 95.87), down 0.01 in search where the cap’s tests were removed along with the code they covered.
  • Known limitation, recorded in the ADR rather than fixed: on 30.2 the weld also hangs against a single-valued edge, whatever its leaves hold. A different defect from the array one, not addressed by fanning out, and free today because a qualified relation is multi-valued by nature and every real edge declares array: true.
  • @lde/search-api-graphql:test is flaky, independently of this branch – Nx now flags it, and its print-sdl case depends on module-graph state (vi.resetModules() then mocking prettier). It failed once here and passed on re-run.

A weld asks whether ONE entry satisfies every condition. An entry whose leaves
hold arrays stands for every combination at once, so it answers the weld with
none of them – and Typesense 30.2, the current stable release, does not answer
at all: the search hangs indefinitely, which reaches a consumer as an
unexplained error.

- refuse a nested field declaring both filterable and array, so a leaf a weld
  can name states one value per entry
- fan an edge out into one entry per combination of its weldable leaves, on the
  framed node before projection, so each leaf still passes through transform,
  folding and the facet companion unchanged
- write a nested identity companion under the arity of its reference, since that
  companion is the leaf a weld actually names; a top-level companion stands for
  the whole document and is unchanged
- bound the fan-out with maxEntries (default 100), per ADR 12: a cartesian
  product over an edge's values is a bound in the data's own units
- pin the engine guarantee against a real container, the array-shaped hang
  included, so a future engine bump reports when it is gone

Measured at 1,000,000 documents: the welded filter answers in 14–19 ms where it
previously never returned, and the fanned-out shape is no slower than the array
shape on 31.0, so the eventual upgrade unwinds nothing. Indexing is ~15 % slower
for ~50 % more entries.

BREAKING CHANGE: a nested field may no longer declare both “filterable” and
“array”; searchSchema refuses it. Declare the leaf single-valued – the
projection now emits one entry per combination.
Three defects found reviewing the fan-out, two of them introduced by it and
each invisible without a live engine.

- restore the already-a-list case when widening a nested leaf. Only Reference
  Types are held to the single-valued rule, and the same path declares the
  fields of the Root Type a local lookup nests, where array and filterable meet
  on an ordinary facet – the leaf was emitted with no type at all, which the
  engine refuses outright
- declare a nested identity companion by what its path yields rather than
  always as a list. Under a single-valued edge the parent is an object, nothing
  flattens, and Typesense rejects the scalar the projection writes against a
  declared list: the whole import fails
- expand every tuple the cap keeps. Stopping mid-expansion left the remaining
  weldable leaves holding their lists, and a single-valued leaf then kept the
  first value and dropped the rest – losing data quietly rather than fanning it
  out

Two existing tests asserted the arity rule inverted, stating that a companion is
always a list and that a single value is what the engine rejects. A live import
says the opposite, so both now read the other way.

An identity on a Reference Type's field is already refused, and a nested
filterable lookup can no longer be an array, so a nested companion never holds
more than one id; no change was needed there. The entry budget is per node
carrying the reference rather than per document, so nested edges compound it –
documented rather than changed.
…e arity 30.2 will not weld

Two limitations found verifying the fan-out against a live engine, neither
visible from a collection definition alone.

- a nested identity companion is typed by the path that reaches it rather than
  by what one entry holds, and Typesense enforces that wherever nothing widens
  the path
- the weld hangs on 30.2 against a single-valued edge whatever its leaves hold,
  which is a different defect from the array one and is not addressed by fanning
  out. It costs nothing today, because a qualified relation is multi-valued by
  nature
…its type

A Root Type reached by a local lookup is nested exactly as a Reference Type is;
it just happens to have a collection of its own elsewhere. Reading that from the
type made the projection treat a locally-nested root as a root, so it wrote a
list where the collection declared a single value – and Typesense rejects the
document at import, for every document carrying such a reference.

- thread the nesting flag from the call site into projectFields, rather than
  deriving it from whether the declaring type declares a class
- declare a nested identity companion by both routes to a list, the field's own
  array as well as a flattening ancestor, as every sibling declaration in that
  function already does
- reject a maxEntries that is not a positive integer. The budget is counted off
  one entry at a time, so a fractional or non-positive cap is never reached and
  the fan-out grows with the data – the unbounded case the cap exists to
  prevent. Compare with >= rather than ==, so no cap can be stepped over
- export DEFAULT_MAX_ENTRIES beside DEFAULT_LABEL_FIELD, and state the number in
  the option's own documentation, so the default is legible without it

Left alone deliberately: a single-valued inline reference still fans out up to
the full budget before keeping one entry. Capping it at one would change which
entry survives when the first projects empty, and the waste is bounded.
…hat survives

Three ways the entry budget took data it was never meant to touch, each verified
against the projection before and after.

- cap only a reference that CAN fan out. The budget bounds the multiplication
  this decision introduces, but it was applied to every inline reference: a
  display-only nesting of 300 entries silently became 100, shortening a list
  that has always been stored whole
- charge it against the entries that SURVIVE, not the framed values considered.
  A run of values the reference type reads nothing from – dirty source data,
  which the emptiness filter exists for – spent the budget and left the real
  referents behind them with nowhere to go, dropping the reference entirely
- narrow a nested companion only where fan-out underwrites it. An identity is
  earned by filterable OR facetable, and only a weldable leaf is ever split, so
  a facetable-only companion kept its first id and dropped the rest – silently,
  because the collection declared it a single value too

The declared type follows the same rule, so the two agree again.

Also drop an unreachable guard: tuplesOf is handed what remains of the budget
and never returns more, so the inner loop cannot overshoot it.

The ADR claimed entries past the cap are “dropped and reported”. There is no
reporting channel and inventing one for data that should never arise is not
worth the seam, so it now says the drop is silent, that truncation follows
declaration order rather than being representative, and why a local lookup is
uncapped.
The fan-out needs a bound – a cartesian product over an edge's own values is no
bound at all – but capping the entries a document stores is not it. By the time
the projection runs every cost has been paid: the CONSTRUCT matched those values
and the endpoint paid for it, they crossed the wire, the subject index holds
them, and framing has materialised them into one node. Capping the product
declines the last and cheapest step while keeping all the expensive ones, and
bounds nothing framing did not already hold.

It also missed the case that predates fan-out entirely: a wide edge, or a
display-only nesting of ten thousand entries, is linear rather than
multiplicative and was never bounded either.

- remove maxEntries, its default, its validation and its export
- remove the budget from the projection, and with it the survivor-charging and
  part-way expansion the cap alone made necessary
- record in ADR 26 why the bound belongs at the framing seam instead, and that
  the fan-out is unbounded until it lands

The bound is tracked separately, where it can be decided on its own evidence:
capping values per leaf where they enter memory makes the product k^(weldable
leaves), and leaf count is a schema constant – a bound in the schema's own units
rather than a number written against the data's.

Lower the branch threshold by 0.01 to match: the tests that raised it existed
only to exercise the cap, and the code they covered has gone with it.
@ddeboer
ddeboer force-pushed the worktree-weld-fanout branch from b57bd33 to 2edf409 Compare September 4, 2026 12:31
…sition

Nothing welds the identity field by name, so fan-out passed over it – but the
flat companion harvested FROM it is the leaf a weld uses to name the endpoint.
An entry whose identity field held three ids therefore stood for three
endpoints at once, and the companion kept the first and dropped the rest: a
filter on either of the others missed a document that genuinely carried it,
with the entry beside it still listing all three.

The same tuple problem the fan-out exists to remove, one field further in, so
it takes the same answer: the identity field is a tuple position like any
weldable leaf, and multiplicity moves to the entry list. Three endpoints now
make three entries holding one endpoint each, and every id stays reachable.

This is what the narrowing in setIdentity always assumed. It justified itself
with "fan-out splits weldable leaves only", which was true of `filterable`
fields and not of the one the companion actually reads.
… drops

Fan-out splits RAW framed values, and two distinct ones can still collapse
downstream – a transform mapping two spellings onto one canonical value, or a
keyed target re-keying two referent IRIs to the same document key. Before
fan-out those values met inside one entry and were deduped there; split across
entries they reached the index and the API as byte-identical duplicates.

- dedupe the projected entries by content, restoring the guarantee the
  pre-fan-out shape had
- document what a single-valued edge now drops. It stores one entry, so an edge
  the graph gave two endpoints indexes the first where the old single entry
  listed both. The old shape was never answerable by a weld, so this is the
  ordinary single-valued rule meeting data that outgrew the declaration – but it
  changes what a filter matches, and nothing can refuse it at startup because
  the declaration is only wrong once the data has more than one value
- correct the justification shared by setIdentity and nestedIdentityFields.
  Both claimed a facetable-only identity is never split, which stopped being
  true when the identity field became a tuple position: it is always split, and
  `filterable` decides only whether the companion is worth the narrower declared
  type. The arity was right; the reason given for it was not

Also fixes the reference docs promising that nothing is dropped, which held only
for an array edge, and counting two consequences before listing three.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A welded co-element filter hangs Typesense when the nested fields hold arrays

1 participant